Skip to content

Fix OverflowError: Python int too large to convert to C long raised during annotation label map creation with custom labels. #548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix OverflowError: Python int too large to convert to C long raised during annotation label map creation with custom labels. #548

wants to merge 1 commit into from

Conversation

@briangow
Copy link
Contributor

@briangow briangow commented Oct 13, 2025 •
edited
Loading

The patch makes the label‑store dtype explicit (int64) and replaces row‑wise .loc assignment with a keyed, vectorized update.

@briangow briangow force-pushed the bg_fix_test_overflow branch from e3a23f4 to 43cf077 Compare October 13, 2025 16:28
@briangow briangow changed the title [DRAFT] Fix annotation test overflow error Fix OverflowError: Python int too large to convert to C long raised during annotation label map creation with custom labels. Oct 13, 2025
@briangow briangow mentioned this pull request Oct 13, 2025
Open
@bemoody
Copy link
Collaborator

bemoody commented Oct 17, 2025 via email

@briangow
Copy link
Contributor Author

@bemoody this is to fix the test error currently seen in #544. My understanding is that this is a known issue when Pandas tries to unify data types. Here is the explanation (along with the full test error):

WFDB’s own constraints keep label_store small (1–49), so it isn’t your annotation values that are “too big.” The overflow comes from pandas’ internal attempt to unify dtypes when a row assignment ends up behaving like an append under the hood. On some platforms/pandas versions, that path tries to convert Python ints through a C long and can raise OverflowError even if your data are reasonable.

=================================== FAILURES ===================================
____________________________ TestAnnotation.test_3 _____________________________

self = <tests.test_annotation.TestAnnotation testMethod=test_3>

    def test_3(self):
        """
        Annotation file with custom annotation types
    
        Target file created with:
            rdann -r sample-data/1003 -a atr > ann-3
        """
>       annotation = wfdb.rdann("sample-data/1003", "atr")

tests/test_annotation.py:183: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
wfdb/io/annotation.py:2028: in rdann
    annotation.set_label_elements(return_label_elements)
wfdb/io/annotation.py:1408: in set_label_elements
    self.convert_label_attribute(contained_elements[0], e)
wfdb/io/annotation.py:1471: in convert_label_attribute
    label_map = self.create_label_map(inplace=False)
wfdb/io/annotation.py:911: in create_label_map
    label_map.loc[i] = self.custom_labels.loc[i]
.venv/lib/python3.9/site-packages/pandas/core/indexing.py:912: in __setitem__
    iloc._setitem_with_indexer(indexer, value, self.name)
.venv/lib/python3.9/site-packages/pandas/core/indexing.py:1933: in _setitem_with_indexer
    self._setitem_with_indexer_missing(indexer, value)
.venv/lib/python3.9/site-packages/pandas/core/indexing.py:2329: in _setitem_with_indexer_missing
    self.obj._mgr = self.obj._append(value)._mgr
.venv/lib/python3.9/site-packages/pandas/core/frame.py:10581: in _append
    other = row_df.infer_objects(copy=False).rename_axis(
.venv/lib/python3.9/site-packages/pandas/core/generic.py:6910: in infer_objects
    new_mgr = self._mgr.convert(copy=copy)
.venv/lib/python3.9/site-packages/pandas/core/internals/managers.py:466: in convert
    return self.apply("convert", copy=copy, using_cow=using_copy_on_write())
.venv/lib/python3.9/site-packages/pandas/core/internals/managers.py:363: in apply
    applied = getattr(b, f)(**kwargs)
.venv/lib/python3.9/site-packages/pandas/core/internals/blocks.py:656: in convert
    blocks = self.split_and_operate(
.venv/lib/python3.9/site-packages/pandas/core/internals/blocks.py:473: in split_and_operate
    rbs = func(nb, *args, **kwargs)
.venv/lib/python3.9/site-packages/pandas/core/internals/blocks.py:675: in convert
    res_values = lib.maybe_convert_objects(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   OverflowError: Python int too large to convert to C long

pandas/_libs/lib.pyx:2619: OverflowError
=========================== short test summary info ============================
FAILED tests/test_annotation.py::TestAnnotation::test_3 - OverflowError: Python int too large to convert to C long
=================== 1 failed, 69 passed in 281.48s (0:04:41) ===================
Error: Process completed with exit code 1.

@bemoody
Copy link
Collaborator

bemoody commented Oct 28, 2025

The problem seems to be related to NPY_PROMOTION_STATE=weak_and_warn; perhaps it is the same or similar to pandas-dev/pandas#60023 (which was closed, understandably, because this was never meant to be supported long-term.)

I'd suggest that we've gotten what benefit we can out of NPY_PROMOTION_STATE, and we can probably just remove it from test.yml. Or we could leave it set only for the test-deb11-i386 job.

As for create_label_map, without understanding this code I'd rather not change it. I'd much rather eliminate the use of pandas and rewrite the logic using standard Python data structures and/or numpy.

@briangow
Copy link
Contributor Author

@bemoody , thanks for looking into this. It does seem like it is probably related to pandas-dev/pandas#60023. I'd be fine with removing NPY_PROMOTION_STATE but if you prefer to keep it only for test-deb11-i386 that is fine also. I can update this PR once you let me know your thoughts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@briangow @bemoody